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Abstract 

The results of a search for pair production of a heavy, top-Hke quark, t', in the decay 
mode t't' — > bW+bW^ — ?> hi^vhi^v are presented. The search is performed with 
a data sample corresponding to an integrated luminosity of 5.0 fb^ in pp collisions 
at a center-of-mass energy of 7 TeV, collected by the CMS experiment at the LHC. 
The observed number of events agrees with the expectation from standard model 
processes, and no evidence of t't' production is found. Upper limits on the production 
cross section as a function of t' mass are presented, and t' masses below 557 GeV/c^ 
are excluded at the 95% confidence level. 



Submitted to Physics Letters B 



*See AppendixfAJfor the list of collaboration members 



1 Introduction 

Since the discovery of the top quark at the Tevatron HI [2l, there have been many searches 
for a possible new generation of fermions. Those searches have found no evidence of new 
fermions beyond the standard model (SM). However, based on present knowledge, there is no 
compelling reason for the number of fermion generations to be limited to three. Additional 
generations of fermions may have a significant effect on neutrino, flavor, and Higgs physics. A 
fourth generation of quarks, t' and b', would allow indirect bounds on the Higgs boson mass 
to be relaxed IS HI, and may result in enough intrinsic matter and anti-matter asymmetry to 
explain the baryon asymmetry of the universe [5J. Therefore, there is continued theoretical and 
experimental interest in the search for a fourth generation fermion [6|. 

Previous direct searches restrict the masses of quarks in the fourth generation, Mt' and Mb', to 
be greater than 404 and 372 GeV/c^, respectively [7, 8|, and the measurement of the Z lineshape 
at the Large Electron-Positron collider excludes a fourth generation of light neutrinos [9H12|. 
At the Large Hadron Collider (LHC), the quantum chromodynamics (QCD) production cross 
section of t't' is expected to be two orders of magnitude larger than that at the Tevatron for 
Mf = 500 GeV/c^ ||T3| . This increase provides an opportunity to explore the possibility of new 
physics w^ith an extended generation of fermions at higher masses. 

We present a search for pair production of a heavy top-like quark in the final state t' — > bW — > 
hiv, where a branching fraction of 100% to bW is assumed and the charged lepton is either an 
electron or a muon. This search is motivated if Mf < Mb' + Mw/ 'which is favored by precision 
electroweak constraints Hlill. The presence of two leptons (dileptons) in the final state helps 
to suppress SM backgrounds, providing a clean environment to search for physics beyond the 
SM. The data sample corresponds to an integrated luminosity of 5.0 fb^ in pp collisions at a 
center-of-mass energy of 7 TeV, collected by the Compact Muon Solenoid (CMS) experiment at 
the LHC during 2011. 

2 CMS detector 

The central feature of the CMS apparatus is a superconducting solenoid, 13 m in length and 
6 m in diameter, which provides an axial magnetic field of 3.8 T. Within the field volume are 
several particle detection systems. Charged particle trajectories are measured by silicon pixel 
and silicon strip trackers, covering < ^ < 27r in azimuth and \ri\ < 2.5 in pseudorapidity, 
where t] = — ln[tan(0/2)] and 9 is the polar angle of the trajectory of the particle with re- 
spect to the counterclockwise proton beam direction. A lead tungstate crystal electromagnetic 
calorimeter and a brass/scintillator hadron calorimeter surround the tracking volume, provid- 
ing energy measurements of electrons and hadronic jets. Muons are identified and measured 
in gas-ionization detectors embedded in the steel flux return yoke of the solenoid. The CMS 
detector is nearly hermetic, allowing energy balance measurements in the plane transverse to 
the beam direction. A tw^o-tier trigger system selects pp collision events of interest for use in 
physics analyses. A more detailed description of the CMS detector can be found elsewhere |[T5|. 

3 Event samples, reconstruction, and preselection 

The data used for this measurement w^ere collected using one of the ee, e^, or ji}i high-px 
double-lepton triggers. Muon candidates are reconstructed using two algorithms that require 
consistent signals in the tracker and muon systems: one matches the extrapolated trajectories 
from the silicon tracker to signals in the muon system (tracker-based muons), and the second 



2 3 Event samples, reconstruction, and preselection 

performs a global fit requiring consistent patterns in the tracker and the muon system (globally 
fitted muons) l|T6ll . Electron candidates are reconstructed starting from a cluster of energy de- 
posits in the electromagnetic calorimeter. The cluster is then matched to signals in the silicon 
tracker. A selection using electron identification variables based on shower shape and track- 
cluster matching is applied to the reconstructed candidates KIT) . Electron candidates within 
AR = y^{Ar])'^ + (A(^)2 < 0.1 from a muon are rejected to remove candidates due to muon 
bremsstrahlung and final-state radiation. Both electrons and muons are required to be isolated 
from other activity in the event. This is achieved by imposing a maximum allowed value of 
0.15 on the ratio of the scalar sum of other transverse track momenta and transverse calorimeter 
energy deposits within a cone of AR < 0.3 around the lepton candidate direction at the origin 
(the transverse momentum of the candidate is excluded), to the transverse momentum of the 
candidate. 

Event preselection is applied to reject events other than those from tt or t't' in the dilepton final 
state. Events are required to have two opposite-sign, isolated leptons (e+e^, e^]A^, or ^^^^). 
Both leptons must have transverse momenta px > 20 GeV/c, and the electrons (muons) must 
have \t]\ < 2.5 (2.4), and must be consistent with a common interaction vertex. In the rare case 
(< 0.1%) of events with more than two such leptons, the two leptons with the highest pj are 
selected. Events with an e+e^ or }i^}i^ pair with invariant mass betw^een 76 and 106 GeV/c^ or 
below 12 GeV/c^ are removed to suppress Drell-Yan (DY) events (Z/7* — )■ ^+£^) as well as low 
mass dilepton resonances. The jets and the missing transverse energy E^*® are reconstructed 
with a particle-flow technique [TS] . The anti-A:T clustering algorithm [ |T9| w^ith a distance pa- 
rameter of 0.5 is used for jet clustering. At least two jets with px > 30GeV/c and \rj\ < 2.5, 
separated by AR > 0.4 from leptons passing the analysis selection, are required in each event. 
Exactly two of these jets are required to be consistent with coming from the decay of heavy 
flavor hadrons and be identified as b jets by the TCHEM b-tagging algorithm [20], which relies 
on tracks with large impact parameters. The Ef^^^ in the event is required to exceed 50 GeV. 

Signal and background events are generated using the MadGraph 4.4.12 [21] and pythia 
6.4.22 im event generators. The samples of tt, DY with M^£ > 50 GeV/c^, diboson (WW, WZ, 
and ZZ only: the contribution from W7 is assumed to be negligible), and single top quark 
events are generated using MadGraph. The DY event samples with M^ < 50 GeV/c^ are gen- 
erated using PYTHIA. The samples of t't' events are generated using MadGraph, but decayed 
using PYTHIA. The t' — > Wb decay is modeled assuming a V-A structure of the interaction. 
Events are then simulated using a GEANT4-based model [23J of the CMS detector, and finally 
reconstructed and analyzed with the same software used to process collision data. The cross 
section for tt production is taken from a recent CMS measurement ||24| . while next-to-leading 
order (NLO) cross sections are used for the remaining SM background samples. The t't' cross 
sections are calculated to approximate next-to-NLO (NNLO) using hathor |25| . 

With the steadily increasing LHC instantaneous luminosity, the mean number of interactions 
in a single bunch crossing also increased over the course of data taking, reaching about 15 
at the end of the 2011 running period. In the following, the yields of simulated events are 
weighted such that the distribution of reconstructed vertices observed in data is reproduced. 
The efficiency for events containing two leptons satisfying the analysis selection to pass at least 
one of the double-lepton triggers is measured to be approximately 100%, 95%, and 90% for the 
ee, e}i, and }i}i triggers, respectively, and corresponding weights are applied to the simulated 
event yields. In addition, b-tagging scale factors are applied to simulated events for each jet, to 
account for the difference between b-tagging efficiencies in data and simulation pO|. 

The observed and simulated yields after the above event preselection are listed in Table [1} in 



which the categories tt -^ £^£^ and DY— ?► £^£^ correspond to dileptonic tt and DY decays, 
including t leptons only when they also decay leptonically All other tt decay modes are in- 
cluded in the category tt — ?► other. The yields are dominated by top-pair production in the 
dilepton final state, and an agreement is observed between data and simulation. The expected 
yields from t't' are also shown for different values of Mt' . 

Table 1: The observed and simulated yields after the preselection described in the text. Un- 
certainties are statistical only. The systematic uncertainties on the simulated yields are given in 
Section [6] For W + jets, where the simulated yields are zero, upper limits are given based on 
the weighted yield, had one of the simulated events passed the preselection. 



Sample 


ee 


m 


e}i 


all 


t'F,Mt/=400GeV/c2 


10.6 ± 0.9 


13.9 ± 1.0 


29 A ± 1.5 


53.9 ± 2.0 


t't',Mt/ = 500GeV/c2 


3.0 ± 0.2 


3.3 ± 0.2 


6.7 ± 0.4 


12.9 ± 0.5 


t'F,Mt/ = 600GeV/c2 


0.9 ± 0.1 


1.0 ±0.1 


2.2 ± 0.1 


4.1 ± 0.2 


tt -^ £+£- 


488 ± 11 


615 ± 12 


1472 ± 19 


2575 ± 25 


tt -> other 


7.2 ± 1.3 


0.5 ± 0.3 


10.5 ± 1.6 


18.2 ± 2.1 


W + jets 


< 1.9 


<1.9 


<1.9 


<1.9 


DY^ £+£- 


2.9 ± 1.5 


1.6 ± 1.0 


0.6 ± 0.5 


5.1 ± 1.8 


Diboson 


0.5 ± 0.1 


1.1 ±0.2 


1.9 ± 0.2 


3.6 ± 0.3 


Single top quark 


15.6 ± 1.0 


19.5 ±1.1 


46.9 ± 1.7 


82.0 ± 2.2 


Total background 


514 ± 11 


637 ± 12 


1532 ± 19 


2683 ± 25 


Data 


510 


615 


1487 


2612 



4 Signal region 

After preselection, the sample is dominated by SM tt events. Since a t' quark is expected to have 
a much larger mass than that of the top quark, variables that are correlated with the decaying 
quark mass can help distinguish t't' events from tt events. The mass of the system defined 
by the lepton and b jet (M^b) from the quark decay is chosen for this purpose. In the decay 
of a given top quark, M^b is less than y M^ — Mw^, where Mt and Mw are the masses of the 
top quark and W boson. In contrast, most t' decays have M^b larger than that value. At the 
reconstruction level, how^ever, there are tw^o ways to combine the two leptons and two b jets in 
each event, giving four possible values of M^,. The minimum value of the four masses (M^"^) 
is found to be a good variable for distinguishing t't' events from tt events, as can be seen in 
Fig-Hi 

The signal region is defined by adding the requirement for the minimum mass of lepton and 
jet pairs to be M^'" > 170GeV/c^. This additional selection reduces the expected number 
of tt events by four orders of magnitude compared with the preselection prediction given in 
Table hi The simulated yields of t't' events are typically reduced by 50%; they are given for 
different values of Mt' in Table |2] 



5 Background estimation 

The dominant source of the migration of background events into the signal region is the misiden- 
tification of b jets and leptons. A misidentified lepton is defined as a lepton candidate not orig- 
inating from a prompt decay, such as a lepton from semileptonic b or c decays, a muon from 



5 Background estimation 



Table 2: The expected yields of t't' events in the signal region for different values of Mf 
Uncertainties are statistical only. 



t't' sample 



Mt'=400GeV/c2 
Mt/ = 500GeV/c2 
Mt/ = 600GeV/c2 



ee jifi eji all 



3.5 ± 0.5 5.5 ± 0.6 11.2 ± 0.9 20.1 ± 1.2 
1.4 ±0.2 1.9 ±0.2 3.3 ±0.2 6.7 ± 0.4 
0.6 ±0.1 0.6 ±0.1 1.3 ±0.1 2.5 ±0.1 



a pion or kaon decay, an unidentified photon conversion, or a pion misidentified as an elec- 
tron. Misidentified b jets are referred to as "mistags", and occur when a non-b jet satisfies the 
b-tagging requirements. 

The background events in the signal region can be divided into the following categories: 

• Category I: events with mistagged b jet(s) and two real leptons; 

• Category II: events with misidentified lepton(s) and two real b jets; 

• Category III: events w^ith tw^o real b jets and two real leptons; 

• Category IV: events with mistagged b jet(s) and misidentified lepton(s). 

For each category, an estimate of the combined yield of ee, efi, and ]i}i events is made. 

To predict the number of events with mistagged b jet(s) (Category I), control regions in data 
are used where events pass all selection requirements except the number of b-tagged jets. The 
number of background events with one mistag, Nimistag/ is estimated from events with one 
b tag. Each event is weighted based on the mistag rate r, for each untagged jet in the event, 
where r, gives the pj- and f/-dependent probability (with a mean of 0.02) for a non-b jet to be 
b-tagged |20|. Where there are no untagged jets passing the selection, the event weight is zero, 
and for each untagged jet i passing the selection the event weight is increased by r, / (1 — r, ) . The 
subtraction of r, in the denominator is necessary to account for non-b jets that were mistagged, 
and are thus missing from the sample of untagged jets. A similar calculation is made using 
events with no b tags to estimate the number of events with two mistags, Nimistags- This time 
a weight of r,/(l — r,) x Vj/ {1 — Yj) is used for each pair of untagged jets passing selection, 
where r, and Yj are the mistag rates for the two untagged jets. The final prediction is obtained 
as Nmistags = ^^imistag — N^2mistags^ which takes into account that Nimistags is counted twice in 
Nimistag- The performance of the method is checked using simulated events, and an under- 
prediction of up to 50% is observed. We therefore assign a large systematic uncertainty, 100%, 
to this prediction. In data, the predicted number of events with mistags in the signal region is 
Nmistags = 0-7 ± 0.3 ± 0.7, where the uncertainties are statistical and systematic, respectively. 
The Category I yield expected from simulation, taken as a cross-check, is 1.0 ± 0.3, and is con- 
sistent with the prediction based on data. 

The background from events with misidentified leptons (Category II) is predicted based on 
the number of events in data with a candidate lepton that can pass only loosened selection 
criteria Il26| . Using a measurement of the fraction of such "loose" leptons that go on to pass 
the selection requirements, the number of misidentified leptons in the event sample can be 
estimated. However, there are no observed data events where one or more of the lepton can- 
didates passes only the loosened selection criteria, resulting in a prediction of O.O^gg events 
where the upper uncertainty corresponds to the prediction of the method, had there been one 
such event. The Category II event yield is also zero in the simulation. 

The simulation is used to predict the number of events w^ith no misidentified b jets or leptons 



(Category III). Selecting only events where both b jets and leptons are well matched to the 
corresponding particles at the generator level, the resulting prediction is 1.0 ± 0.7 where the 
uncertainty is statistical. The systematic uncertainty is small in comparison (Section |6]), so the 
total uncertainty is also 0.7. 

The contribution of events from Category IV is assumed to be negligible and is covered by both 
the Category I and Category II predictions. Since the Category II prediction is zero, there is no 
possibility of double-counting. 

6 Systematic uncertainties 

The systematic uncertainty on the overall selection efficiency is dominated by the uncertainty 
on the b-tagging efficiency. This uncertainty is 15% for b jets w^ith pj > 240 GeV/c, and 4% for 
b jets with px ^ 240GeV/c [20]. Other uncertainties include those on trigger efficiency (2%), 
lepton selection (2%), and jet and E^'** energy scale (8%) [27J. These four sources combine 
to yield a 19% relative uncertainty on the overall selection efficiency. There is a further 2.2% 
uncertainty on the luminosity measurement ||28| . 

The systematic uncertainty on the background estimate is dominated by the uncertainty on the 
estimate of events with mistagged b jets from data (100%), and by the lack of selected events in 
the loose-lepton control region. The systematic uncertainties on these sources of background 
are included in the summary of background predictions given in Table |3] 

7 Results and summary 

The number of expected events from background processes is 1.8 ± 1.1, and one event is ob- 
served in the eji channel. There is thus no evidence for an excess of events above SM expecta- 
tions. A summary of the observed and predicted yields is presented in Table |3] 

The simulated distribution of M^" from background processes is compared with the data in 
Fig. hi where the expected distribution for a t't' signal w^ith M^r = 450 GeV/c^ is also shown. The 
simulated background yields in the signal region are scaled so that they match the yields esti- 
mated from control regions in data, as given in Table |3] Outside the signal region the simulated 
background yields are taken without rescaling. 

Table 3: Summary of the predicted background yields and the measured yield in data. Statis- 
tical and systematic uncertainties are combined. 



Sample 


Yield 


Category I (from data) 
Category II (from data) 
Category III (simulated) 


0.7 ±0.8 

0+0-4 

1.0 ±0.7 


Total prediction 


1.8 ±1.1 


Data 


1 



Finally, 95% confidence level (CL) upper limits on the production cross section of t't' as a func- 
tion of t' mass are set, using the CLg method 1291 130| . in w^hich a log-normal model of nuisance 
parameter integration is assumed. 

The limit calculation is based on the information provided by the observed event count com- 
bined with the values and the uncertainties of the luminosity measurement, the background 
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Figure 1: Comparison between the data and the simulated background for M^™. The signal 
region is defined by M^"^ > 170 GeV/c^. The simulated background yields in the signal region 
are scaled so that they match the yields estimated from control regions in data, as given in 
Table |3] Outside the signal region the simulated background yields are taken without rescaling. 
One event is observed in the signal region. The expected distribution for a t't' signal with 
Mf = 450 GeV/c^ is also shown. 



prediction, and the fraction of t't' events expected to be selected. This fraction (the overall se- 
lection efficiency) is taken as the product of efficiency, acceptance, and the branching fraction 
for simulated signal events, and is given in Table |4] for different values of Mt'. The calculated 
limits are shown in Table [5] and Fig. [2] 

In summary, assuming a branching fraction of 100% for t' — > bW, the expected and observed 
95% CL lower bounds on the t' mass are 547 and 557 GeV/c^, respectively, from the analysis 
of a data sample of pp collisions at \/s = 7 TeV, corresponding to an integrated luminosity of 
5.0fb"\ 

Table 4: Overall selection efficiency in simulated events for different t' masses. The branching 
fraction of 6.5% for the dilepton decay mode of t't' is included. The uncertainties are calculated 
using the systematic uncertainty of 19% from Section [6J 



1 1' sample 



Mt/ = 350GeV/c2 
Mt'=400GeV/c2 
Mt'=450GeV/c2 
Mt/ = 500GeV/c2 
Mt/ = 550GeV/c2 
Mt/ = 600GeV/c2 



EffxAccxBr(%) 



0.16 ± 0.03 
0.29 ± 0.06 
0.35 ± 0.07 
0.41 ± 0.08 
0.48 ± 0.09 
0.54 ± 0.10 



Table 5: The approximate NNLO theoretical cross section of t't' production assuming standard 
QCD couplings [25], and the expected and observed 95% CL upper limits on the production 
cross section of t't', for different t' masses. 



Mt'(GeV/c2) 350 


400 


450 


500 


550 


600 


Theoretical cross section (pb) 3.20 


1.41 


0.62 


0.33 


0.17 


0.09 


Expected limit (pb) 0.53 


0.29 


0.24 


0.21 


0.18 


0.16 


Observed limit (pb) 0.47 


0.26 


0.22 


0.18 


0.16 


0.14 



Q.10 



t 

Q. 
Q. 



CMS, 5.0 fb"^ at Vs=7 TeV 

■ ■ Theory (HATHOR) [25] 

95% CL expected limits 

95% CL observed iimits 

^B Expected iimits + 1a 
Expected iimits ± 2a 




350 400 450 500 550 600 

M,, (GeV/c^) 

Figure 2: The 95% CL upper limits on the production cross section of t't' as a function of t' 
mass. The observed (expected) 95% CL lower bound on Mt' is 557 (547) GeV/c^. 
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